[draft] port to `wasm32-unknown-unknown` & web by makemeunsee · Pull Request #8607 · helix-editor/helix

makemeunsee · 2023-10-24T23:08:34Z

Hi Helix team,

As mentioned in a discussion, here's a draft PR for my work to have (a subset of) Helix fully bundled as web app.

The README of the new crate helix-web lists current limitations and known issues.

Let me know if you'd like me to point out things here, or if you prefer to have a look first.

Also I'm looking now at re-enabling parsing with tree-sitter, I could compile some parsers to wasm so I'll try to embed them, instead of hot loading them, and use them. Not sure if I should push that work to the same branch though, it would make this PR even messier.

Demo

archseer · 2023-10-25T06:42:43Z

@pascalkuthe continuing from #8588 (reply in thread), I think tokio added wasm support over the last year, running in a single thread in a service worker

archseer · 2023-10-25T06:43:22Z

+#[cfg(target_arch = "wasm32")]
+use instant::Instant;
+#[cfg(not(target_arch = "wasm32"))]
+use std::time::Instant;


Rather than adding this conditional everywhere, let's re-export instant in the root of the helix-core crate, then use that everywhere else

archseer · 2023-10-25T06:45:01Z

        insert_final_newline(doc, view);
    }

+    #[cfg(feature = "dap_lsp")]


Let's separate out two feature flags, dap and lsp. That's what I did here master...gui

I originally tried to rebase your gui branch but it was too much outdated. So I redid some of the work you did there in the first, and took a shortcut for these 2 features, knowing neither would work any time soon on wasm. But yeah, they should be distinct, I kinda regret not splitting them in the first place...

archseer · 2023-10-25T06:46:55Z

    Ok(language)
 }

+#[cfg(not(target_arch = "wasm32"))]


All of this git/build handling seems like it could be split into a separate file/module then gated behind a single conditional

I think this is how you could add a WASM entrypoint for grammars too: add an alternative loader implementation that just uses an internal list of compiled-in grammars

yes that's exactly what I'm attempting now; on the Helix side it looks quite doable, it's all the tree-sitter-* libs which will be more tedious to bring to wasm.

I got syntax highlighting for rust, toml and regex to work, but it's very hacky:

hacked tree-sitter-c2rust so it uses instant instead of std::time

hacked C code of tree-sitter-* crates, for wasm compilation + dependency on hacked tree-sitter-c2rust; not big changes but still tedious and I dont see a way to automate the process. Maybe c2rusting the language libs, but I have no experience with it.

It's nice to see it working but it's quite a mess, so I'll keep it on the side for now.

Why do we need to use tree-sitter-c2rust rather than https://github.com/tree-sitter/tree-sitter/blob/master/lib/binding_web/README.md ? I think we should look at how the tree-sitter playground functions since it runs grammars via wasm (and it lazy loads them instead of compiling them into the main binary)

https://tree-sitter.github.io/tree-sitter/playground

Oh that's interesting, I hadnt spotted they had the playground in Rust on wasm too.

It's puzzling, as tree-sitter actually compiles to wasm32-unknown-emscripten, but not to wasm32-unknown-unknown. And wasm32-unknown-emscripten is not compatible with wasm-bindgen: https://rustwasm.github.io/docs/wasm-bindgen/reference/rust-targets.html?highlight=emscripten#other-web-targets.

I'll take a look, it would be so much better this way.

Ok so the playground is a Rust web server, serving emscripten wasm binaries and regular JS code. The hot loading is js/emscripten code, see https://github.com/tree-sitter/tree-sitter/blob/7c0cee70f55fdfe1667890d372441a85f3127ee6/lib/binding_web/binding.js#L1024-L1042

The Rust bindings compile to linux and wasm-emscripten, but not to wasm-unknown.

I dont see an easy way to interop between a wasm32-unknown app and wasm32-emscripten libs. Creating Rust bindings for tree-sitter-js and then writing a facade/adapter so Helix code can abstract over the native and web bindings? I got a headache just thinking about it 😅

Open to any ideas!

archseer · 2023-10-25T08:24:01Z

Testing this out, xterm.js actually supports truecolor (but I guess doesn't advertise correctly). Setting :set true-color true I'm able to use the default theme. You probably want to add another override similar to windows for wasm:

helix/helix-term/src/lib.rs

Lines 25 to 28 in 5b74eaa

    
           #[cfg(windows)] 
        
           fn true_color() -> bool { 
        
               true 
        
           }

archseer · 2023-10-25T08:27:26Z

+      - name: Install wasm-pack
+        uses: jetli/wasm-pack-action@v0.4.0
+        with:
+          version: 'latest'
+


The action does so little it would be nice to just replace with the official invocation: https://rustwasm.github.io/docs/wasm-bindgen/wasm-bindgen-test/continuous-integration.html#github-actions

I picked this one as they advertise much faster (seconds vs. minutes) set up times, but we can use the official one sure.

- name: Install run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh

From the docs should be equally as fast, it figures out the target architecture and downloads the release (rather than cargo-install-ing and compiling from source)

makemeunsee · 2023-10-26T15:42:28Z

I've just rebased onto master, while it's still manageable.

I'll try to ack the various comments over the next days, and point out of few other things to discuss.

but regular build is broken, see TODOs in Cargo.toml files and rust-lang/cargo#12821

still many warnings and some functionalities are just commented out, not feature gated

broken `cargo build`, fix by uncommenting lines helix-editor#43-46 of helix-tui/Cargo.toml

broken `cargo build`, fix by uncommenting lines helix-editor#43-46 of helix-tui/Cargo.toml & tokio features

`cargo build --no-default-features --target wasm32-unknown-unknown`: works for helix-tui and dependencies

* feature gated vcs * select instant::Instant or std::time::Instant or tokio::time::Instant * redraw & idle timers, wip

can now run on wasm: ``` let app = Application::new( Args::default(), Config::default(), helix_core::config::default_syntax_loader(), ); ```

preliminary xterm integration & xterm backend

- xterm rs bindings (partial, just enough for now) - crossterm xterm backend (copied impl, not integrated) - reduced tokio deps

currently broken

archseer · 2023-10-27T01:44:43Z

To get this merged I'd split the PR into parts:

Adding feature flags for lsp and dap, enabling by default
Adding wasm conditionals, and adding a wasm target to the CI build
helix-web, loader, tree-sitter changes etc.

With first two out of the way it would make the PR easier to rebase since the changes would be mostly scoped to helix-web

makemeunsee · 2023-10-27T10:20:00Z

To get this merged I'd split the PR into parts:
* Adding feature flags for lsp and dap, enabling by default

* Adding wasm conditionals, and adding a wasm target to the CI build

* `helix-web`, loader, tree-sitter changes etc.
With first two out of the way it would make the PR easier to rebase since the changes would be mostly scoped to helix-web

Sounds like a plan!

makemeunsee · 2023-10-27T16:05:34Z

Created #8638

restored loading full original languages.toml (actual fix somehow) simplified build manual filtering of grammars

archseer reviewed Oct 25, 2023

View reviewed changes

makemeunsee force-pushed the wasm32 branch from f4e994b to 5ba54ec Compare October 26, 2023 15:38

makemeunsee added 22 commits October 26, 2023 20:59

helix-core compiles to wasm32

8cc20a4

wip helix view wasm32

a42389e

horrible feature gating

ea954e3

helix-view compiles to wasm32

f2a647b

but regular build is broken, see TODOs in Cargo.toml files and rust-lang/cargo#12821

fix for regular, global build

840a0be

still many warnings and some functionalities are just commented out, not feature gated

better defaults; helix-tui wasm32 compilation

b83a156

broken `cargo build`, fix by uncommenting lines helix-editor#43-46 of helix-tui/Cargo.toml

prev commit amend; helix-tui wasm32 compilation

e185dc7

broken `cargo build`, fix by uncommenting lines helix-editor#43-46 of helix-tui/Cargo.toml & tokio features

cargo build works everywhere

4a09826

`cargo build --no-default-features --target wasm32-unknown-unknown`: works for helix-tui and dependencies

restored dap&lsp for non wasm build

d026459

project builds as lib to wasm32; coarse doc

c7a8bc1

minor steps torward wasm32 app

d6c14ba

fixing runtime step by step

facc9dd

more wasm runtime compat

eec0b19

* feature gated vcs * select instant::Instant or std::time::Instant or tokio::time::Instant * redraw & idle timers, wip

more wasm runtime compat

46e1d09

can now run on wasm: ``` let app = Application::new( Args::default(), Config::default(), helix_core::config::default_syntax_loader(), ); ```

crude webapp embedding helix as wasm32

bc111a3

poc xterm integration

8e86a7a

app runs! event loop too, somehow

4982249

preliminary xterm integration & xterm backend

full integration, wip

3639d8a

- xterm rs bindings (partial, just enough for now) - crossterm xterm backend (copied impl, not integrated) - reduced tokio deps

wip crossterm/xterm integration

4083931

currently broken

terminal.size() for wasm32 at last; to be tested

fea0252

it works O_o

4ae3331

wasm32 build fix & GH pipeline

778e1e4

makemeunsee added 12 commits October 26, 2023 20:59

no history for GH pages

a361a3c

cleanup & increased compliance with baseline

e81fc0e

search in selection bug fix

22325d0

actually compatible with node 20

440017b

actually compatible with node 20

38994d2

issue with nucleo

978a1b8

README update

cd5336b

fancy font

241260d

correct rs-xterm-js

879951c

wasm cosmetics: true color on, fallback font

b4d25cc

rebased on master

47751ce

dependency bump

f2dabe6

makemeunsee force-pushed the wasm32 branch from 5ba54ec to f2dabe6 Compare October 26, 2023 19:01

embed themes in wasm

68a1f9d

makemeunsee mentioned this pull request Oct 27, 2023

Feature gating for eventual wasm compilation #8638

Open

makemeunsee added 11 commits October 29, 2023 01:36

pickers are broken on wasm32, disabled

7c127ae

crude buffer read/write using Web Storage API

b147fe8

web storage write/read fixed; tutor embedded

ee32fae

'lightweight' embedding of some TS languages

86b0dc9

limited langs, CI fix maybe

8356662

CI fix maybe

3c604df

CI debug logs

ec7d154

avoid pointless rebuilds

bb4d58c

fetch grammars for wasm32 builds too

e8f3ae1

build fix + optimisation

708bce5

prev. commit broke TS parsing

3b4b5a7

restored loading full original languages.toml (actual fix somehow) simplified build manual filtering of grammars

Rudxain mentioned this pull request Jun 18, 2025

add grammar feature flag #13795

Closed

Uh oh!

Uh oh!

Conversation

makemeunsee commented Oct 24, 2023 • edited by archseer Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

archseer commented Oct 25, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

archseer commented Oct 25, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

makemeunsee commented Oct 26, 2023

Uh oh!

archseer commented Oct 27, 2023

Uh oh!

makemeunsee commented Oct 27, 2023

Uh oh!

makemeunsee commented Oct 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

makemeunsee commented Oct 24, 2023 •

edited by archseer

Loading